Stress Test for Bert and Deep Models: Predicting Words from Italian Poetry

In this paper we present a set of experiments carried out with BERT on number Italian sentences taken from poetry domain. The are organized the hypothesis very high level difficulty in predictability at three levels linguistic complexity that intend to monitor: lexical, syntactic and semantic level. To test ran version 80 - for total 900 tokens – mostly extracted first half last century. Then alternated canonical non-canonical versions same sentence before processing them DL model. We used then newswire domain containing similar structures. results show model is highly sensitive presence However, DLs also word frequency local non-literal meaning compositional effect. This apparent by preference predicting function vs content words, collocates infrequent phrases. paper, focused our attention use subword units done vocabulary words.